DOCUMENT RESUME 



ED 395 383 



EA 027 573 



TITLE 

INSTITUTION 



REPORT NO 
PUB DATE 
NOTE 

PUB TYPE 



Measuring Up: Questions and Answers 3 bout State Roles 
in Educational Accountability. 

Office of Educational Research and Improvement (ED) , 
Washington, DC. Programs for the Improvement of 
Practice. 

PIP-89-820 
Nov 88 

13p . ; For the complete report of the OERI Study Group 
that is summarized here, see ED 299 706. 

Reports - Evaluative/Feasibility (142) — Guides - 
Non-Classroom Use (055) 



EDRS PRICE MF01/PC01 Plus Postage. 

DESCRIPTORS *Accountabi 1 i ty ; Data Collection; Educational 

Assessment; Elementary Secondary Education; 

Evaluation Criteria; Government Role; Guidelines; 
Information Disseminat ion; Information Util ization; 
^Performance; *State Government; State School 
District Relationship 

ABSTRACT 

This booklet provides questions and answers that 
point to the increasing interest that states have in developing, 
shaping, and evaluating the success of schools. The booklet 
highlights the analysis and recommendations of representat ives of 10 
states (California, Colorado, Connecticut, Florida, Georgia, 

Illinois, Massachusetts , Minnesota, New Jersey, and South Carolina) 
who were assembled by the United States Department of Education to 
study recent changes in accountability policies. Recommendations 
include: (1) make the purpose of the system clear to schools and to 
the public; (2) coordinate state and local data systems to ensure 
collecting data of maximum usefulness to both; (3) establish a 
blue-ribbon, independent body to oversee the system and recommend 
adjustments; (4) use multiple indicators of performance, not just 
test scores, to provide a fair and integrated picture of schooling; 

(5) consider a broad range of help to encourage both absolute high 
performance and comparative high performance; (6) when states 
intervene in cases of poor school performance, they should proceed in 
fair and well-defined stages; and (7) make school and district 
performance data public. The booklet also contains data collected by 
the Council of Chief State School Officers (CCSSO) in the fall of 
1987, including a summary of the key dimensions of the 50 states' 
performance-accountability systems. Every state must answer the 
following six questions when designing an accountability system: Who 
designs the system? What data will the state or district collect? At 
what levels should data be collected and used? To whom should the 
data be reported and in what. form? Should schools be compared and, if 
so, in what ways? and Should consequences be linked to the findings 
of accountability reports, and, if so, which ones? (LMI) 
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More and more citizens want to know in concrete terms what they are 
receiving for their hard-earned tax dollar investments in education. Gover- 
nors , legislators, State leaders, and blue-ribbon committees are being 
called upon to provide more financial support for public schools. They 
naturally want to know whether this support yields a return. They, there- 
fore, need better information about student performance and school out- 
comes to hold sch(X)ls accountable for results and improvement. 

Tferry Peterson, Study Group Chair 

All of the facts in this booklet point to the increasing interest States have 
in developing, shaping, and evaluating ways of measuring the success of 
schools. This policy brief highlights the analysis and recommendations of 
representatives of 10 States who were assembled by the U.S. Department 
of Education to study recent changes in accountability policies. In addi- 
tion, this booklet contains data collected by the Council of Chief State 
School Officers (CCSSO) in the fall of 1987. 
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MEASURING UP: 

Questions and Answers About State 
Roles in Educational Accountability 



How well are schools doing? And how do we know? 



States try to measure school performance and report to the 
public. 

Since 1981, States have increased their aid to local school districts 
dramatically — by a national average of 21 percent. States want to know 
the results of their spending in keeping with a general trend to collect and 
report performance data on public services. The comparative economic 
well-being of the State and Nation depends on increased levels of skills 
and literacy in the work force. States need to know if today’s students are 
going to measure up. They need information to improve education effec- 
tively. 

With performance data in hand, States are better able to design policies 
that do improve local schools. Performance data can identify which schools 
need help, suggest what kind they need, and determine when they need it. 
They can provide information upon which to waive regulations, reward ex- 
cellence, or increase technical assistance or funding. These data have 
proved to be powerful policy tools that significantly influence what schools 
plan and teach. 



Why do we need to know? 



Tb hold schools accountable and help improve their performance. 



Reliable information is the key to better performance in education as it is 
in business and government. In the past, most people judged their schools 
on the basis of personal knowledge. First-hand experience, community 
satisfaction, and a responsive principal were enough to convince most 
people that their schools were doing fine. 

Today, States are developing, or requiring local districts to design, systems 
for collecting, reporting, and using objective data on student and school 
performance. The issue is “hot” because being held accountable to tax- 
payers, training the future work force, and improving the caliber of schools 
are important political issues. 

State systems are in flux; 30 of the 50 States report that they are consider- 
ing changes. Commonly, these “systems” evolve in incremental changes 
over time. New reports may be issued, and consequences may be linked to 
data the State has collected for years. Once established, these systems 
provide continuing information and therefore influence how future school 
q success will be judged. 
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What would educators gain from an accountability system? 



Better information with which to plan and operate. 

Information about the results of schooling can help both policymakers and 
educators judge the overall merit of schools. When information is detailed 
enough to identify strengths and weaknesses within a school, it can be 
used by local educators to demonstrate the merits and plan the improve- 
ment of local education programs. Data enable teachers and principals to 
adjust curricula, instruction, and teaching materials and to monitor suc- 
cess at meeting goals. In addition. States can offer additional policy 
rewards and incentives for performance. 



Then why do these systems involve political tensions? 



They affect who will be held accountable to t.'hom for what . 

They may become lightning rods for tension between State and local offi- 
cials. Local districts fear losing control to the State; educators fear being 
judged unfairly; and everyone fears improper emphasis upon what is easi- 
ly measured at the expense of what is most important When high stakes 
consequences, such as large cash awards or district consolidations, are 
linked to performance, incentives exist for deception or manipulation of 
data in order for a school to look good. 

The resolution of these tensions lies in finding a politically acceptable 
agreement between the public and educators about the purpose of the sys- 
tem and the fairness of its operation. While policymakers and the public 
need an honest accounting, professional educators need assurance that 
their work will be judged fairly. The Study Group recommends creation of 
a blue-ribbon, independent body to oversee the ac^ustment and improve- 
ment of this process. 



What are the toughest choices? 



Decisions that involve complex trade-offs. 

Creating an accountability system requires tough choices on issues involv- 
ing complex trade-offs. Many important dimensions of school performance 
cannot be easily measured. Nonetheless, policymakers seek clarity of pur- 
pose, mutuality of understanding, and fairness of operation to prevent sub- 
version of a system that local officials might consider unfair. A sound 
educational . ccountability system, therefore, must consider both: 
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■ State accountability and local autonomy. 

■ The need to know how well schools are doing in overall performance 
(excellent, mediocre, or poor) and the need to know other information 
about specific areas (such as writing skills, knowledge of foreign lan- 
guages, minority success, or teacher quality) to improve diagnoses and 
plan improvement. 

■ The need for Statewide comparability and the need for local ownership. 

■ The usefulness, thoroughness, and fairness of collecting and analyzing 
new data and the the burden of paperwork, time, and effort it takes to 
do so. 

■ The economy of easy assessment methods, such as nationally normed 
standardized tests of basic skills and the importance of other measures 
of school performance, such as assessments of problem-solving skills or 
participation in the arts. 

■ A State’s desire to assess local schools and help them improve and con- 
sideration of the State budget, staff expertise, political climate, and 
technology available to do so. 



What does the State have to do? 

Answer six key questions. 

Public disclosure of performance data is likely to help improve schools 
when data are consistent with State and local goals, results are reported to 
schools and the public in a useful and timely way, and local districts have 
the resources and ability to act upon what they learn. 

An accountability system can backfire if test scores are all that count or if 
school comparisons are unfair. Although no two States have devised iden- 
tical systems, every State must answer these six key questions when 
designing a system: 



States currently collect 
many kinds of data, 
including: 



student achievement 
teat scores *44) 

high school completion 1 

rates < 44 > j 



student 

attendance *-42) 



student competency 
test scores ■ 28) 



student status ailcr 
high school < 27) 



other *47 1 . 
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Who designs the system? 

Most States report that they retain primary responsibility for the design, 
implementation, and use of their accountability reporting systems them- 
selves (see table, page 6). Two States, Minnesota and Vermont, require 
local districts to do so, and nine report that they share the responsibility 
with local districts. It is generally agreed that State systems must be coor- 
dinated with district and school accountability systems. 

What data will the State or district collect? 

Ideally, data should: 

■ measure the central and most important features of schooling; 

■ measure what is taught or considered important; 

■ provide information that can be acted upon; 

■ focus on school-level information (not just State averages); 

■ allow for fair comparisons; and 

■ maximize the usefulness and minimize the burden of collecting data. 
These data can be presented as: 

■ National, State, district, and school-building averages; 
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Data of school performance 
may include: 

!ndtcatt*rs at Retails 
• J raau ai : o dr -• pout r ; it e s ; 

Teacher, student attendance; 

Student wr.'.;r.2 samples: 

Achievement and competency test 
scores; 

Students’ problem-solving skills: 
Participation m arts and 
extracurricular activities: 

SAT ’A CT *co re s : 

National Assessment of Education ‘ii 
Progress scores; 

| Stuuent *=i at u s a 1 to r h 1 jr'n sc n .xii: 
j Km mover g at i s fact; on: 

Pm ;n*t t:.-.v:ird State Hoard a : : 

Course enrolments 'in advanced 
: - : act* me n t . ; re i trn i a n gu a lv ? . 

I lienees. ,ir.< ; 

Pi Trent .1 -talents meet inc State 

’ ■ * : i vi • r s : t v • t ra nee rvu.u » re rr. cat-*. 

hmiccu;r< a "out /Wiry 
Amount ana value of homewors; 

Class size; 

Placement in academic tracks: 

Average icacr.cr salary; 

Teacher qualifications; 

Per pupil spending; 

Implementation of State reforms; 

Indicators about Context 
Community support and wealth; 
Student characteristics of race, wealth, 
language, parents* education. 



■ Changes over time: since last year; since 5 years ago; since major legis- 
lation or appropriations; or 

■ Distribution of achievement over subgroups: racial/ethnic groups; voca- 
tional education or college-bound students; groups with limited English 
proficiency; males and females. 

Despite the widespread collection of varied data, only 24 States say they 
collect and use data based on an integrated set of indicators. 



At what levels — school, district, State, or 
individual student — should data be collected 
and used? 

The Study Group recommends publicly reporting performance data at the 
school-building level, as well as at the district and State levels. To do so, 
some tests, like promotion or high school graduation exams, must be taken 
by every student, whereas others can be given to a sample of students. 
State and local collection of data should be coordinated. 



To whom should the data be reported, and in 
what form? 

Some States create comprehensive “report cards” on every school and 
release them directly to the public, press, and schools; others report raw 
data to districts, who themselves report it to the schools and the public. 
Still other States only make performance data available to the public on 
request. 
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Forty-three States make 
performance data public. 

Segments shr-iv the smallest unit 
assessed and the number of States 
reporting data about that unit to the 
public. 




A number of States offer workshops to help schools and the press under- 
stand the format and interpret the findings of State reports. The Study 
Group recommends making special efforts to report and interpret data 
clearly, in forms that are intellectually honeet and understandable to 
parents, teachers, and the public. The decisions a State makes about who 
is going to get information, in what form, and with what provision for in- 
terpretation are related to one another and to the purpose and effective- 
ness of the system. 

According to the Council of Chief State School Officers (CCSSO), States 
now vary in reporting data. Twenty-four States do not report school-level 
achievement test data to the media, parents, or the public. On the other 
hand, 38 States report these data to principals, 36 to districts, 19 to 
teachers, 18 to the media, 17 to parents, and 17 to the public. 



Should schools be compared? And, if so, in 
what ways? 

A controversial and important decision each State makes is whether to 
compare the performance of schools, and if so, upon what dimensions. 
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Should schools be compared: 



■ With local or State-specified goals? 

■ With past performance (demonstrating improvement)? 

■ With test norms? 

■ With all other schools (as rankings)? 

■ With similar schools? 

If the latter, what are the important aspects of similarity: free lunch en- 
rollment, racial/ethnic composition, per pupil spending, or parents’ educa- 
tion? 

Currently, according to CCSSO, 22 States report comparisons of schools* 
test results. Comparing schools on other dimensions of performance and 
against multiple indicators is more complex. Although 45 States collect 
contextual/demographic information, only 23 use this data to interpret 
student achievement or performance on other indicators. 

California and South Carolina have pioneered the creation of “comparison 
bands” in which the performance of individual schools is shown in relation 
to that of similar schools. California reports each school’s performance 
with relation to a “floating band” of the 160 schools across the State near- 
est it in demographic characteristics. South Carolina compares each 
school’s performance to that of others with similar student bodies and cal- 
culates each school’s expected scores on the basis of previous test results. 
When such bands are not used, it is difficult to distinguish the results of 
good schooling from those of social privilege. On the other hand, when they 
are used, there is the possibility that they can institutionalize lower expec- 
tations for disadvantaged students. The goal is a system that recognizes 
social context but holds high expectations for all students. 

Should consequences (such as rewards or 
sanctions) be linked to the findings of 
accountability reports? And, if so, which ones? 

Public disclosure of performance data has in itself powerful consequences 
for education. Some States feel the intrinsic value of improved education 
requires little or no additional policy incentives. Others hope effective ac- 
countability systems will enhance local autonomy by relieving successful 
schools of some State regulation. Most States also use collected data to 
provide rewards and sanctions as policy incentives for improvement. 

States that formally recognize outstanding school performance need to 
communicate their selection criteria clearly. Some States recognize schools 
for both absolute high performance (ranking near the top of all schools in 
the State) and comparative high performance (ranking near the top of 
schools with similar student bodies or improving significantly over past 
performance). Only one-third of the States that recognize outstanding 
schools report that doing so is automatically triggered by high school as- 
sessment results, without an additional nomination or application process. 

As a last resort, six States may intervene in local district management in 
response to low school performance. This may mean designating 
“bankruptcy” or “impairment” and ultimately either replacing the local su- 
perintendent and board or requiring consolidation with a nearby school 
district. 



What States are doing with 
performance data 

According to a study by CCSSO. States 

Recognize one of the following . . . 46 

Teachers 46 

Students 36 

Schools 33 

Districts 14 



Require local improvement 

planning 14 

Allocate funds for remedial 

programs 9 

Intervene in local management . . 6 

Provide financial incentives 5 

Withhold funus 3 

I Waive regui at ions d 



KEY DIMENSIONS OF THE 50-STATE PERFORMANCE ACCOUNTABILITY SYSTEMS 
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Indicator 

System 


Test 


Public Report 








Policy 

Links 


State 


Level 


Type 


School 


District State 


Compare Context 


Alabama 


State 


No 


Both 


Yes 


Yes 


Yes 


Yes 


No 


No 


Alaska 


None 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


Arizona 


State 


No 


Achievement 


Yes 


Yes 


Yes 


Yes 


No 


No 


Arkansas 


State 


No 


Both 


No 


Yes 


Yes 


Yes 


Yes 


Yes 


California 


Mixed 


Yes 


Both 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Colorado 


Mixed 


Yes 


Achievement 


No 


Yes 


Yes 


Yes 


Yes 


No 


Connecticut 


State 


Yes 


Achievement 


No 


Yes 


Yes 


Yes 


Yes 


Yes 


Delaware 


State 


No 


Achievement 


Yes 


Yes 


Yes 

Yes 1 


Yes 


No 


No 


DC 


State 1 


No 


Both 


Yes 


Yes 


Yes 


No 


Yes 


Florida 


State 


Yes 


Both 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Georgia 


State 


No 


Both 


Yes 


Yes 


Yes 

Yes 1 


Yes 


Yes 


Yes 


Hawaii 


State 1 


Yes 


Both 


Yes 


Yes 


Yes 


Yes 


Yes 


Idaho 


State 


No 


Achievement 


No 


No 


No 


No 


No 


No 


Illinois 


Mixed 


Yes 


Achievement 


Yes 


Yes 


Yes 


Yes 


No 


Yes 


Indiana 


State 


Yes 


Both 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Iowa 


None 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


Kansas 


State 


Yes 


Achievement 


No 


Yes 


Yes 


No 


Yes 


No 


Kentucky 


State 


Yes 


Both 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Louisiana 


State 


No 


Both 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Maine 


State 


No 


Achievement 


Yes 


Yes 


Yes 


Yes 


No 


No 


Maryland 


State 


No 


Both 


Yes 


Yes 


Yes 


Yes 


No 


No 


Massachusetts 


State 


No 


Both 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Michigan 


State 


No 


Achievement 


Yes 


Yes 


Yes 


Yes 


No 


Yes 


Minnesota 


Local 


No 


Achievement 


No 


Yes 


Yes 


Yes 


No 


Yes 


Mississippi 


State 


Yes 


Both 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Missouri 


State 


No 


Both 


No 2 


Yes 


Yes 


No 


No 


No 


Montana 


None 


No 


Achievement 3 


1 No 


No 


No 


N/A 


N/A 


N/A 


Nebraska 


None 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


Nevada 


State 


Yes 


Both 


No 


Yes 


Yes 


Yes 


No 


No 


New Hampshire State 


No 


Achievement 


No 

_ 


No 


No 


Yes 


No 


No 
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Indicator 


Teat 


Public Report 






Policy 


State 


Level 


System 


Type 


School 


District 


State 


Compare Context 


Links 


New Jersey 


State 


No 


Both 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


New Mexico 


Mixed 


Yes 


Both 


No 


No 


Yes 


Yes 


Yes 


Yes 


New York 


State 


Yes 


Both 


Yes 


Yes 


Yes 


Yes 


No 


Yes 


North Carolina 


State 


No 


Both 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


North Dakota 


None 


No 


Achievement 3 


No 2 


No 2 


Yes 


Yes 


No 


No 


Ohio 


Mixed 


Yes 


Both 


Yes 


Yes 


Yes 


No 


No 


No 


Oklahoma 


State 


No 


Achievement 


No 


Yes 


Yes 


Yes 


No 


Yes 


Oregon 


Mixed 


Yes 


Both 


No 


No 


Yes 


No 


No 


No 


Pennsylvania 


State 


Yes 


Both 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Rhode Island 


State 


Yes 


Achievement 


Yes 


Yes 


Yes 


Yes 


No 


No 


South Carolina 


State 


Yes 


Both 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


South Dakota 


State 


No 


Achievement 


No 


No 


No 


No 


No 


No 


Tennessee 


State 


No 


Both 


No 


Yes 


Yes 


Yes 


No 


No 


Texas 


State 


Yes 


Both 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Utah 


Mixed 


Yes 


Both 


No 


No 


Yes 


Yes 


Yes 


No 


Vermont 


Local 


Yes 


Competency 


No 


No 


N/A 


N/A 


No 


No 


Virginia 


State 


No 


Both 


No 


Yes 


Yes 


Yes 


No 


Yes 


Washington 


State 


Yes 


Achievement 


Jin 


Yes 


Yes 


Yes 


Yes 


Yes 


West Virginia 


Mixed 


Yes 


Both 


No 


Yes 


Yes 


Yes 


No 


No 


Wisconsin 


Mixed 


No 


Both 


No 


No 


Yes 


No 


No 


No 


Wyoming 


State 


No 


Achievement 


No 


No 


Yes 


No 


No 


No 


Totals 


S=35 


Yes=23 


A=18 


Yes=25 


Yes=37 


Yes=4J 


Yes=38 


Yes=21 


Yes=25 




L=2 


No=25 


C=1 


No=23 


No=ll 


No=4 


No=8 


No=26 


No=22 




M=9 


N/A=3 


Both=29 


N/A=3 


N/A=3 


N/A=4 


1 N/A=5 


N/A=4 


N/A=4 




N=5 




N/A=3 















SOURCE: Council ofChicf State School Officers 1987 Survey and related State documents. 

1 The District of Columbia and Hawaii each operate a single system in which the State and the district arc the same. 

2 Missouri and North Dakota send school— (plus North Dakota district)— level data to parents but not to the press. 

3 Montana and North Dakota offer local districts the option of using a State achievement test. 
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Recommendations 



Be Clear 
Be Fair 
Be Open 

On the basis of their experience, Study Group members 
recommend: 

1. Make the purpose of the system clear to schools and to the 
public. 

2. Coordinate State and local data systems to ensure collect- 
ing data of maximum usefulness to both. 

3. Establish a blue-ribbon, independent body to oversee the 
system and recommend adjustments. 

4. Use multiple indicators of performance (not just test scores) 
to give a fair and integrated picture of schooling. 

5. Consider a broad range of help— incentives, rewards, and 
relief from regulation — to encourage both absolute high per- 
formance (ranking near the top of all schools in the State) and 
comparative high performance (schools making significant 
improvement or ranking near the top of schools with similar 
student bodies). 

6. If State intervention is considered appropriate in cases of 
poor school performance, proceed in fair and well-defined 
stages. Give local districts ample support and opportunity to 
improve on their own before the State intervenes. 

7. Make school and district performance data public. 
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For further information 



The complete report of the OERI Study Group is available for $4.00 each 
from the Superintendent of Documents, U.S. Government Printing Office, 
Washington, DC 20402 (request stock number 065-000-00352-0). Send 
check or money order payable to “Superintendent of Documents.” 
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